
            <!DOCTYPE html>
            <html lang="en">
            <head>
                <meta charset="UTF-8">
                <title>【Prometheus】Relabeling机制</title>
            </head>
            <body>
            <a href="https://andyoung.blog.csdn.net">原作者博客</a>
            <div id="content_views" class="markdown_views prism-atom-one-light">
                    <svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
                        <path stroke-linecap="round" d="M5,0 0,2.5 5,5z" id="raphael-marker-block" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0);"></path>
                    </svg>
                    <p><img src="https://i-blog.csdnimg.cn/blog_migrate/6f51fb8116127aca0400c5151147cf6d.jpeg#pic_center" alt=""></p> 
<h2><a id="PrometheusRelabeling_3"></a>Prometheus的Relabeling机制</h2> 
<p>在Prometheus所有的Target实例中，都包含一些默认的Metadata标签信息。可以通过Prometheus UI的Targets页面中查看这些实例的Metadata标签的内容：</p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/810fbb2ce307d2356f29bead18801419.png" alt="实例的Metadata信息"></p> 
<p>默认情况下，当Prometheus加载Target实例完成后，这些Target时候都会包含一些默认的标签：</p> 
<ul><li><code>__address__</code>：当前Target实例的访问地址<code>&lt;host&gt;:&lt;port&gt;</code></li><li><code>__scheme__</code>：采集目标服务访问地址的HTTP Scheme，HTTP或者HTTPS</li><li><code>__metrics_path__</code>：采集目标服务访问地址的访问路径</li><li><code>__param_&lt;name&gt;</code>：采集任务目标服务的中包含的请求参数</li></ul> 
<p>上面这些标签将会告诉Prometheus如何从该Target实例中获取监控数据。除了这些默认的标签以外，我们还可以为Target添加自定义的标签。</p> 
<p>一般来说，Target以<code>__</code>作为前置的标签是在系统内部使用的，因此这些标签<mark><strong>不会被写入到样本数据中</strong></mark>。不过这里有一些例外，例如，我们会发现所有通过Prometheus采集的样本数据中都会包含一个名为instance的标签，该标签的内容对应到Target实例的<code>__address__</code>。 这里实际上是发生了一次标签的重写处理。</p> 
<p>这种发生在采集样本数据之前，对Target实例的标签进行重写的机制在Prometheus被称为Relabeling。</p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/949d69fcf64d4a244d1e2a45af2a4ddb.png" alt="Relabeling作用时机"></p> 
<p>Prometheus允许用户在采集任务设置中通过relabel_configs来添加自定义的Relabeling过程。</p> 
<ul><li>Relabel用来重写target的标签</li><li>每个Target可以配置多个Relabel动作，按照配置文件顺序应用</li><li>Target包含一些内置的标签（以’__'开头），都可以用于relabel，在relabel时未保留，<strong>内置标签将被删除</strong></li></ul> 
<h2><a id="relabel_33"></a>relabel流程</h2> 
<pre><code>Target（[source_label,…]） -&gt; relabel -&gt;  Target （[target_label,…]）
</code></pre> 
<h2><a id="Relabel_39"></a>Relabel的配置</h2> 
<pre><code> [ source_labels: '[' &lt;labelname&gt; [, ...] ']' ]
 [ separator: &lt;string&gt; | default = ; ]
 [ target_label: &lt;labelname&gt; ]
 [ regex: &lt;regex&gt; | default = (.*) ]
 [ modulus: &lt;uint64&gt; ]
 [ replacement: &lt;string&gt; | default = $1 ]
 [ action: &lt;relabel_action&gt; | default = replace ]
</code></pre> 
<h3><a id="Relabelaction_51"></a>Relabel的action</h3> 
<table><thead><tr><th align="center">ACTION</th><th align="center">Regex匹配</th><th align="center">操作对象</th><th align="left">重要参数</th><th align="left">描述</th></tr></thead><tbody><tr><td align="center">keep</td><td align="center">标签值</td><td align="center"><strong>Target</strong></td><td align="left">源标签、regex</td><td align="left">丢弃指定源标签的标签值没有匹配到regex的target</td></tr><tr><td align="center">Drop</td><td align="center">标签值</td><td align="center"><strong>Target</strong></td><td align="left">源标签、regex</td><td align="left">丢弃指定源标签的标签值匹配到regex的target</td></tr><tr><td align="center">labeldrop</td><td align="center">标签名</td><td align="center">Label</td><td align="left">Regex</td><td align="left">丢弃匹配到regex 的标签</td></tr><tr><td align="center">labelkeep</td><td align="center">标签名</td><td align="center">Label</td><td align="left">Regex</td><td align="left">丢弃没有匹配到regex 的标签</td></tr><tr><td align="center"><strong>Replace</strong></td><td align="center">标签值</td><td align="center">Label名+值</td><td align="left">源标签、目标标签、替换（值）、regex（值）</td><td align="left">更改标签名、更改标签值、合并标签</td></tr><tr><td align="center">hashmod</td><td align="center">无</td><td align="center">标签名+值</td><td align="left">源标签、hash长度、target标签</td><td align="left">将多个源标签的值进行hash，作为target标签的值</td></tr><tr><td align="center">labelmap</td><td align="center">标签名</td><td align="center">标签名</td><td align="left">regex、replacement</td><td align="left">Regex匹配名-&gt;replacement用原标签名的部分来替换名</td></tr></tbody></table> 
<h3><a id="replaceactionaction_63"></a><strong>replace是缺省action，可以不配置action</strong></h3> 
<p>使用labeldrop 和labelkeep Relabel后需要注意保证metrics+labels唯一</p> 
<h3><a id="markregex__replacement_mark_69"></a><mark>regex 和 replacement 参数详解</mark></h3> 
<p>regex 正则匹配字段是标签名、标签值具体看上面的表格， Replacement会用到了正则捕获组（需要自行补充相关知识）来替换相应的值（标签名、值）</p> 
<h3><a id="_75"></a>如何查看源标签</h3> 
<p>从prometheus-》status-》service Discovery</p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/767ef9a9f6b302f22b80672452acefd4.png" alt="如何查看源标签"></p> 
<h3><a id="target_83"></a>过滤target</h3> 
<ul><li>使用keep，保留标签值匹配regex的targets</li></ul> 
<pre><code class="prism language-yaml"><span class="token key atrule">scrape_configs</span><span class="token punctuation">:</span>
 <span class="token punctuation">-</span> …
 <span class="token punctuation">-</span> <span class="token key atrule">job_name</span><span class="token punctuation">:</span> <span class="token string">"cephs"</span>
    <span class="token key atrule">relabel_configs</span><span class="token punctuation">:</span>
      <span class="token punctuation">-</span> <span class="token key atrule">action</span><span class="token punctuation">:</span> keep
        <span class="token key atrule">source_labels</span><span class="token punctuation">:</span>
          <span class="token punctuation">-</span>  __address__
        <span class="token key atrule">regex</span><span class="token punctuation">:</span>  ceph01.* 
</code></pre> 
<p>relabel结果可以在Prometheus网页的status/ Service Discovery中查看</p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/bde5d2088cbd5b7478c85362fc5d4c60.png" alt="过滤target"></p> 
<ul><li>使用drop，丢弃匹配regex的targets</li></ul> 
<pre><code class="prism language-yaml"><span class="token key atrule">scrape_configs</span><span class="token punctuation">:</span>
 <span class="token punctuation">-</span> …
 <span class="token punctuation">-</span> <span class="token key atrule">job_name</span><span class="token punctuation">:</span> <span class="token string">"cephs"</span>
    <span class="token key atrule">relabel_configs</span><span class="token punctuation">:</span>
      <span class="token punctuation">-</span> <span class="token key atrule">action</span><span class="token punctuation">:</span> drop
        <span class="token key atrule">source_labels</span><span class="token punctuation">:</span>
          <span class="token punctuation">-</span>  __address__
        <span class="token key atrule">regex</span><span class="token punctuation">:</span>  ceph01.*
</code></pre> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/d0868af5bca92373b82352d458837f8f.png" alt="使用drop，丢弃匹配regex的targets"></p> 
<h3><a id="_123"></a>删除标签</h3> 
<p>将标签名为job的标签删除</p> 
<pre><code class="prism language-yaml"><span class="token key atrule">scrape_configs</span><span class="token punctuation">:</span>
 <span class="token punctuation">-</span> …
  <span class="token punctuation">-</span> <span class="token key atrule">job_name</span><span class="token punctuation">:</span> <span class="token string">"cephs"</span>
    <span class="token key atrule">relabel_configs</span><span class="token punctuation">:</span>
      <span class="token punctuation">-</span> <span class="token key atrule">regex</span><span class="token punctuation">:</span> job
        <span class="token key atrule">action</span><span class="token punctuation">:</span> labeldrop
</code></pre> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/4e23504cf944330482ef096a26057698.png" alt="labeldrop"></p> 
<p>labelKeep和labeldrop不操作’__’开头的标签，要操作需要先改名</p> 
<h3><a id="label_144"></a>修改label名</h3> 
<p>使用replace将<strong>scheme</strong>标签改名为protocol</p> 
<pre><code class="prism language-yaml"><span class="token key atrule">scrape_configs</span><span class="token punctuation">:</span>
  <span class="token punctuation">-</span> …
  <span class="token punctuation">-</span> <span class="token key atrule">job_name</span><span class="token punctuation">:</span> <span class="token string">"cephs"</span>
    <span class="token key atrule">relabel_configs</span><span class="token punctuation">:</span>
      <span class="token punctuation">-</span> <span class="token key atrule">source_labels</span><span class="token punctuation">:</span>
          <span class="token punctuation">-</span> __scheme__
        <span class="token key atrule">target_label</span><span class="token punctuation">:</span> procotol
</code></pre> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/0ec2ec963ef75e2824308be946cbc777.png" alt="修改label名"></p> 
<p>这里可以是多个source_labels，只有值匹配到regex，才会进行替换</p> 
<ul><li>使用labelmap，将原始标签的一部分转换为target标签，这一功能replace无法实现</li></ul> 
<pre><code class="prism language-yaml"><span class="token key atrule">scrape_configs</span><span class="token punctuation">:</span>
  <span class="token punctuation">-</span> …
  <span class="token punctuation">-</span> <span class="token key atrule">job_name</span><span class="token punctuation">:</span> <span class="token string">"sd_file_mysql"</span>
    <span class="token key atrule">file_sd_configs</span><span class="token punctuation">:</span>
      <span class="token punctuation">-</span> <span class="token key atrule">files</span><span class="token punctuation">:</span>
        <span class="token punctuation">-</span> mysql.yml
        <span class="token key atrule">refresh_interval</span><span class="token punctuation">:</span> 1m
    <span class="token key atrule">relabel_configs</span><span class="token punctuation">:</span>
      <span class="token punctuation">-</span> <span class="token key atrule">action</span><span class="token punctuation">:</span> labelmap
        <span class="token key atrule">regex</span><span class="token punctuation">:</span> (.<span class="token important">*)(address)(.*)</span> 
        <span class="token key atrule">replacement</span><span class="token punctuation">:</span> $<span class="token punctuation">{<!-- --></span><span class="token number">2</span><span class="token punctuation">}</span>
</code></pre> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/3c64134c1e77ffd261a64e644695c630.png" alt="labelmap"></p> 
<h3><a id="label_184"></a>修改label值</h3> 
<p>配置k8s服务发现</p> 
<pre><code class="prism language-yaml"><span class="token key atrule">scrape_configs</span><span class="token punctuation">:</span>
  <span class="token punctuation">-</span> …
  <span class="token punctuation">-</span> <span class="token key atrule">job_name</span><span class="token punctuation">:</span> <span class="token string">"sd_k8s_nodes"</span>
    <span class="token key atrule">kubernetes_sd_configs</span><span class="token punctuation">:</span>
      <span class="token punctuation">-</span> <span class="token key atrule">role</span><span class="token punctuation">:</span> node
        <span class="token key atrule">bearer_token_file</span><span class="token punctuation">:</span> bearer_token
        <span class="token key atrule">tls_config</span><span class="token punctuation">:</span>
          <span class="token key atrule">ca_file</span><span class="token punctuation">:</span> ca.crt
        <span class="token key atrule">namespaces</span><span class="token punctuation">:</span>
          <span class="token key atrule">names</span><span class="token punctuation">:</span>
            <span class="token punctuation">-</span> default
        <span class="token key atrule">api_server</span><span class="token punctuation">:</span> https<span class="token punctuation">:</span>//master01<span class="token punctuation">:</span><span class="token number">6443</span>
</code></pre> 
<p>服务发现完成后，默认node的port是10250，会无法取得数据，同通过relabel修改标签.</p> 
<pre><code class="prism language-yaml">    <span class="token key atrule">relabel_configs</span><span class="token punctuation">:</span>
      <span class="token punctuation">-</span> <span class="token key atrule">source_labels</span><span class="token punctuation">:</span>
         <span class="token punctuation">-</span> __address__
        <span class="token key atrule">regex</span><span class="token punctuation">:</span> (.<span class="token important">*)\:</span><span class="token number">10250</span>
        <span class="token key atrule">replacement</span><span class="token punctuation">:</span> <span class="token string">"${1}:10255"</span>
        <span class="token key atrule">target_label</span><span class="token punctuation">:</span> __address__
</code></pre> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/033a734482a9f1b0897220dce28e1d0e.png" alt="replacement"></p> 
<h3><a id="_220"></a>多标签合并</h3> 
<p>标签合并，可以将多个源标签合并为一个目标标签，可以取源标签的值，也可以进行hash，用户target分组</p> 
<ul><li>在文件服务发现中，将标签filename=“mysql.yml” 和sd_type="file"合并为sd=”file;mysql.yml”，标签值使用分号连接</li></ul> 
<pre><code class="prism language-yaml"><span class="token key atrule">scrape_configs</span><span class="token punctuation">:</span>
<span class="token punctuation">-</span> …
  <span class="token punctuation">-</span> <span class="token key atrule">job_name</span><span class="token punctuation">:</span> <span class="token string">"sd_file_mysql"</span>
    <span class="token key atrule">file_sd_configs</span><span class="token punctuation">:</span>
      <span class="token punctuation">-</span> <span class="token key atrule">files</span><span class="token punctuation">:</span>
        <span class="token punctuation">-</span> mysql.yml
        <span class="token key atrule">refresh_interval</span><span class="token punctuation">:</span> 1m
    <span class="token key atrule">relabel_configs</span><span class="token punctuation">:</span>
      <span class="token punctuation">-</span> <span class="token key atrule">source_labels</span><span class="token punctuation">:</span>
          <span class="token punctuation">-</span> sd_type
          <span class="token punctuation">-</span> filename
        <span class="token key atrule">separator</span><span class="token punctuation">:</span> ;
        <span class="token key atrule">target_label</span><span class="token punctuation">:</span> sd
</code></pre> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/d8acfcf17cfa430cb5933b94af754022.png" alt="多标签合并"></p> 
<ul><li>将多个标签的值进行hash，形成一个target标签，只要target标签一致，则表示源标签一致，可以用来实现prometheus的负载均衡</li></ul> 
<pre><code class="prism language-yaml"><span class="token key atrule">scrape_configs</span><span class="token punctuation">:</span>
  <span class="token punctuation">-</span> …
  <span class="token punctuation">-</span> <span class="token key atrule">job_name</span><span class="token punctuation">:</span> <span class="token string">"sd_file_mysql"</span>
    <span class="token key atrule">file_sd_configs</span><span class="token punctuation">:</span>
      <span class="token punctuation">-</span> <span class="token key atrule">files</span><span class="token punctuation">:</span>
        <span class="token punctuation">-</span> mysql.yml
        <span class="token key atrule">refresh_interval</span><span class="token punctuation">:</span> 1m
    <span class="token key atrule">relabel_configs</span><span class="token punctuation">:</span>
      <span class="token punctuation">-</span> <span class="token key atrule">action</span><span class="token punctuation">:</span> hashmod
        <span class="token key atrule">source_labels</span><span class="token punctuation">:</span>
          <span class="token punctuation">-</span> __scheme__
          <span class="token punctuation">-</span> __metrics_path__
        <span class="token key atrule">modulus</span><span class="token punctuation">:</span> <span class="token number">64</span>
        <span class="token key atrule">target_label</span><span class="token punctuation">:</span> hash_id
</code></pre> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/61fb9a788f9d856b91e11ec5e676f96c.png" alt="多标签合并"></p> 
<h2><a id="_273"></a>完整案例</h2> 
<p>以下是一个完整的relabel案例，这个案例包括</p> 
<ul><li>根据标签值过滤target</li><li>合并标签值，并进行正则匹配</li><li>修改标签名</li><li>直接添加标签名</li></ul> 
<p>这个案例说明源标签是可以重复使用的</p> 
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/4db551c415f466ff2a885a5bd6cf7f27.png" alt="完整案例"></p>
                </div>
            </body>
            </html>
            